Towards an Architecture Model for Emotion Recognition in Interactive Systems: Application to a Ballet Dance Show

In the context of the very dynamic and challenging domain of affective computing, we adopt a software engineering point of view on emotion recognition in interactive systems. Our goal is threefold: first, developing an architecture model for emotion recognition. This architecture model emphasizes multimodality and reusability. Second, developing a prototype based on this architecture model. For this prototype we focus on gesturebased emotion recognition. And third, using this prototype for augmenting a ballet dance show. We hence describe an overview of our work so far, from the design of a flexible and multimodal emotion recognition architecture model, to a presentation of a gesture-based emotion recognition prototype based on this model, to a prototype that augments a ballet stage, taking emotions as inputs. INTRODUCTION Whereas interaction among human beings heavily relies on emotional communication, interaction with computers yet lacks the emotional side of communication and is hence unnatural to us. The field of affective computing [18] emerged in the beginning of the nineties to address this issue: giving the computer a way to assess a user’s affective state (i.e. be able to decode the affective signals a human might send – emotion as an input) and to provide affective feedback (sending signals that a human could decode – emotion as an output). Emotion recognition is a young field that does not yet benefit from dedicated models. Our goal is threefold: providing with an extendable, reusable, and multimodal emotion recognition architecture model; building a gesture-based emotion recognition system prototype based on this architecture model; and finally, emotionally augmenting a ballet dance show. As for the last point, we aim at providing a better artistic experience for the audience. Though not being emotion synthesis, this part relies on some emotion-inducing phenomena. In this paper, we will provide an overview of our work in progress: we will first describe our architecture model and the gesture-based emotion recognition prototype. We will then present the Shadoz application for virtually augmenting a ballet dance show. All of these works are in progress, and we will present the identified limitations of our current system. 1 A GENERIC ARCHITECTURE MODEL FOR EMOTION RECOGNITION In order to integrate emotion recognition in any interactive system, we define a generic software architecture model. Few generic models exist for emotion recognition [4] [16]. We define a flexible and multimodal architecture model, the “emotion branch”. We limit ourselves in the definition of this model to emotion interpretation of physical cues from the user, such as face expressions, vocal characteristics, ANS, gesture and posture. We don't aim at covering each of these modalities but rather provide a model for implementing a multimodal emotion recognition system. The emotion branch model is a collection of software components functioning as black boxes. Those components can emit data flows and subscribe to data flows from other components, not unlike [1]. Our model allows consideration of any of the emotional communication channels: facial expressions, voice modulations, physiological cues, etc. Its flexibility emphasizes reusability, modifiability, and possibilities of distributing the recognition process onto several processes or machines. In this part, we first present the key components that define the emotion branch and then explain how the emotion branch can be plugged into canonical software architecture for interactive systems.